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Abstract Body 


Background / Context: 

Improving student achievement and closing the achievement gap is at the forefront of America’s 
education policy agenda. There is nationwide attention to alternatives to the traditional school 
calendar in order to improve academic performance; recent federal funding opportunities 
including the Race to the Top initiative and School Improvement Grant programs include focus 
on underperforming schools providing additional instructional time for students. Recent reviews 
of the literature found that although designs are generally weak for making causal inferences, 
extending school time is mixed, but can be an effective way to support student achievement, 
particularly for disadvantaged students and when attention is paid to how time is used (Redd et 
al., 2012, Patall, Cooper, and Allen, 2010). 

The Massachusetts Expanded Learning Time (ELT) initiative was established in 2005 with state 
resources to allow a selected number of schools to explore a redesign of their respective 
schedules and add time to their day or year. Participating schools must expand learning time by 
at least 300 hours per year and focus on improving student outcomes in core academic subjects, 
broaden enrichment opportunities, and improve instruction by adding more planning and 
professional development time for teachers. Three cohorts of schools have been awarded 
ongoing implementation grants since 2006-2007; the staggered nature of the ELT initiative 
means that at the end of 2010-1 1, participating schools had completed five, four, and three years 
of implementation (Cohorts 1, 2, and 3, respectively). 

ELT implementation is hypothesized to lead to a number of desired outcomes for students and 
teachers. Theory suggests that as a result of ELT, students may become more engaged in school 
because of additional enrichment and learning opportunities, develop better communication 
skills due to more time with teachers and peers, and be less likely to engage in disruptive 
behavior because of less idle time. Teachers may find their teaching experience to be more 
rewarding because they have more adequate time to plan, prepare and instruct, as well as earn 
higher pay and have additional opportunities to develop meaningful relationships with students. 
Ultimately, student achievement may improve as a result of additional learning time. 

Purpose / Objective / Research Question / Focus of Study: 

The Massachusetts Department of Elementary and Secondary Education (ESE) and Abt 
Associates, with grant funding from the U.S. Department of Education’s Institute for Education 
Sciences (IES), completed a five year study of the ELT initiative to examine three primary 
research questions: 

1) How has expanded learning time been implemented in schools that receive ELT grants? 

2) What are the outcomes of expanded learning time for schools, students, and teachers? 

3) What is the relationship between implementation and outcomes? 

Setting: 

The ELT initiative is a state-wide initiative administered by the Massachusetts Department of 
Elementary and Secondary Education. 
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Population / Participants / Subjects: 

This study included 24 elementary, middle, and K-8 ELT schools that were funded by the state 
and 25 matched comparison schools. ELT schools are predominantly under-performing 
(identified for improvement, corrective action, or restructuring based on student achievement 
results), with 50% or more of students from low income families and minority populations and at 
least 10% have special education designations. The non-ELT matched comparison schools were 
selected using a multi-tiered one-to-one matching process (see Exhibit 1) to be as similar as 
possible to ELT schools’ grade span, district, prior student achievement, student demographics, 
and teacher qualifications. Exhibit 2 summarizes comparisons between ELT and comparison 
school characteristics in the last pre-ELT year. 

Intervention / Program / Practice: 

The Massachusetts Expanded Learning Time (ELT) initiative provides grants to selected schools 
to redesign their schedules by adding 300-plus instructional hours to the school year to improve 
outcomes, broaden enrichment opportunities, and provide teachers with more planning and 
professional development time. 

Research Design: 

As random assignment of schools or students to ELT was not feasible, this study’s impact 
analysis relies upon a strong longitudinal quasi-experimental design: a comparative interrupted 
time-series approach that leverages pre-program data and data from matched comparison schools 
to produce estimated effects representing differences between ELT and comparison schools 
beyond what one might expect given pre-program measures and other secular initiatives 
affecting all schools. Analyses of non-academic outcomes rely on cross-sectional survey data 
and use multi-level models that produce estimates of differences between ELT and comparison 
schools to approximate what would have happened in the absence of ELT. 

Data Collection and Analysis: 

Each year, longitudinal student-level MCAS and other extant data for both ELT and matched 
comparison schools are analyzed. These datasets includes student-level achievement scores 
(from English Language Arts, Match, and Science exams), student-level demographic and 
behavior variable, and for 8 th grade students, a self-report questionnaire about time spent on 
homework, computer usage, and future plans. 

During the spring of each year, teachers and students in ELT and matched comparison schools 
were surveyed to examine their perceptions and attitudes. Site visits were conducted each year 
to ELT schools to interview key stakeholders and conduct select focus groups about program 
implementation and time use. District staff, community partners, and state-level stakeholders 
were also interviewed each year to provide key contextual information. 

Analysis of extant achievement data uses a comparative interrupted time series design that 
leverages pre-ELT data, school, and year fixed effects when estimating the effect of ELT. This 
design is among the strongest quasi-experimental designs available, although its analyses are 
non-experimental. Since schools and their students were not randomly assigned to ELT 
participation, results cannot be attributed solely to ELT. The interrupted time series design, use 
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of matched comparison schools and statistical controls, and rigorous model specification, taken 
together, are capable of yielding credible and robust estimates of program impacts. Analysis of 
survey data measuring non-academic outcomes uses multi-level models that regress student and 
teacher outcomes on a school level variable indicating ELT status, school characteristics, 
student/teacher demographic variables, and a dummy variable for each matched pair of schools. 
Models are appropriately adjusted for clustering of students within classrooms and schools and 
teachers within schools. One of the study’s key contributions was to integrate implementation 
and outcomes data using an index based on principles of effective ELT operation; this 
implementation index provides a measure of fidelity that can be used both to understand school- 
level implementation and to explore relationships between implementation and outcomes. 

Findings / Results: 

Implementation 

• There is strong evidence that the ELT study schools implemented many core ELT 
elements during the 2010-11 school year, both in terms of additional time available for 
instruction, academic support, and enrichment and supports for teachers’ use of that time. 
However, schools varied considerably in their respective efforts to implement the core 
components (see Exhibit 3). 

• On average, almost five of the nearly eight hours of a typical school day were allocated to core 
academics (English Language Arts (ELA), math, science, and social studies) (see Exhibit 4). 

o Overall, the majority of time in an ELT school day was allocated to ELA, followed by 
math, then science and social studies. 

• There is substantial variation across ELT schools in the level and approach to implementation. 

o Some comparison schools are implementing core elements of ELT. 

• Measuring different aspects of time use is challenging: collecting information on a prototypical 
student in a given grade level may or may not reveal how students are supported by the ELT 
initiative and definitions of various activities/time uses are not consistent across schools. 

Outcomes 

• More ELT teachers were satisfied with time available for instmction and planning, and reported 
that they spend sufficient instructional time with students. Fewer ELT teachers reported that 
student academic performance and homework completion rates were problem areas. 

• However, more teachers in ELT schools reported that teacher and staff fatigue, as well as student 
fatigue, were problems in their respective schools. More ELT students reported being tired in 
school and fewer ELT students looked forward to school, liked being in school, reported that all 
of their classes were important to them, and liked the length of their school day. 

• Generally, there were no statistically significant effects of ELT on student achievement (see 
Exhibit 5). 

o There were no statistically significant effects of ELT in the first, second, or third years of 
implementation on student achievement, but in the fourth year of implementation 5 th 
grade ELT students performed significantly higher on science tests than would be 
expected in the absence of ELT. 
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Relationship between Implementation and Outcomes 


• Descriptive analysis linking the level of implementation in ELT schools and student achievement 
outcomes indicate no clear patterns or meaningful relationships. 

• Exploratory analysis of differential effects of ELT in higher- versus lower- implementing schools 
indicates minimal heterogeneity in the effect by the level of ELT implementation. 

• The school reform landscape is dynamic; each year, more schools (outside of this ELT initiative) 
appear to be expanding the amount of time in their school year as well as implementing reforms 
consistent with the core ELT components. 

Conclusions: 

Despite the demonstrable progress ELT schools have made to implement core components of the 
initiative, those implementation efforts have not yet consistently translated the additional time 
into the content, strategies, or support that in turn yield overall student performance. One 
potential explanation for the limited average effect of ELT on student achievement is that the 
patterns of implementation differ as much across the ELT schools as between the ELT and 
comparison schools, highlighting the variation in ELT across the initiative. The fact that such 
variation exists in the initiative’s fifth year illustrates both the complexity inherent in large-scale 
efforts to transform low-performing schools and the variation that inevitably results from flexible 
interventions that can be shared to fit individual schools’ needs. 

Given the considerable variability in ELT implementation across schools, the considerable 
variation in schools’ initial motivation to become ELT schools, and ongoing changes in the 
education reform landscape during the same period, it may not be surprising that the study has 
yet to find significant student achievement gains attributable to this initiative. Expansion of time 
for learning is increasingly perceived as an important vehicle for improving educational 
outcomes for disadvantaged students in Massachusetts and elsewhere — if the additional time is 
used well. This study was able to assess the quantity and allocation of time, but did not measure 
the quality of instruction, enrichment, and other activities made possible by the additional time, 
and clearly, the quality of such activities is also important, although given the variability of 
activities measurement would be difficult. 
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Appendices 
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Appendix B. Tables and Figures 


Exhibit 1: Tiered Approach to Selection of Comparison Schools 

Tier 1: Highest priority matching variables 

• ELA Composite Performance Index (CPI), 

• Math Composite Performance Index (CPI), 

• Aggregate ELA Adequate Yearly Progress (AYP), 

• Aggregate Math Adequate Yearly Progress (AYP) 

Tier 2: High priority matching variables 

• ELA Accountability Status, 

• Math Accountability Status 

Tier 3: Medium priority matching variables 

• Student enrollment, 

• Percent minority (i.e., percent non-White), 

• Percent limited English proficiency (LEP), 

• Percent low-income, 

• Percent special education 

Tier 4: Lower priority matching variables 

• Percent male, 

• Percent of teachers in core academic subjects who are highly qualified 
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Exhibit 2: School-Level Characteristics of ELT Schools and Matched Comparison Schools in 
the Final Year Prior to ELT Implementation 



Final Pre-ELT Year 





Difference 


Actual ELT 

Actual 

(in effect size 

Characteristic 

Mean 

MC Mean 

units) 

Student Enrollment 

462 

558 

-0.47 A 

ELA CPI 

73 

75 

-0.22 

Math CPI 

60 

63 

-0.19 

Percent Minority 

56% 

51% 

0.19 

Percent Limited English Proficient 

12% 

13% 

-0.12 

Percent Low Income 

72% 

61% 

0.64 A 

Percent Special Education 

17% 

19% 

-0.49 A 

Percent Male 

52% 

52% 

-0.21 

Percent of Teachers in Core Academic Subjects who 
are Highly Qualified 

97% 

94% 

0.37 A 


EXHIBIT READS: In the year prior to the implementation of ELT, the average enrollment was 462 in ELT schools and 558 
in matched comparison schools. The difference corresponds to an effect size of 0.47 standard deviations. 

A A indicates an effect size of at least 0.25 standard deviations. 

Note: Means shown are averages across schools within each respective group. 

Source :MA ESE website (http://profiles.doe.mass.edu/). 

Sample: Full sample of 49 schools in the outcomes study, including 24 ELT and 25 matched comparison schools in 2009-10 
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Exhibit 3: ELT School Implementation Pie Charts, by Index Criterion, 2010-11 
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EXHIBIT READS: On the implementation index, School J scored a Level 3 for the Academic Focus criterion. 

Notes: School IDs (letters) have been randomly assigned in order to present school-level implementation data anonymously. Pie charts are organized in descending order of total score 
from left to right. 

The color scheme for this Exhibit is deliberately distinct from other exhibits to enhance readability in both color and black and white. 

Sources: Abt Associates ’ Interviews of MA ELT and Matched Comparison School Principals, Spring 2011; Abt Associates ’ Surveys of MA ELT School Teachers, Spring 2011 
Sample: 18 ELT schools with the exception of Criterions 2 and 3, for which the sample is 17 ELT schools. 
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Exhibit 4: Amount and Percentage of Time Allocated to Core Academics in ELT Schools Per 

Day, by Grade, 2010-11 

Panel 1 



□ ELA aMath mScience □ Social Studies 


Panel 2 

ELT Overall ELT 5th ELT 8th 



EXHIBIT READS (Panel 1): In spring 2011, across all ELT schools, 1 hour and 45 minutes per day were allocated to 
instruction in ELA. 

EXHIBIT READS (Panel 2): In spring 2011, across all ELT schools, 37 percent of the time allocated to core academics was 
allocated to instruction in ELA. 

Source: Abt Associates ’ Interviews of MA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 1 -8 th grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 th grades. 
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Exhibit 5: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation Year 

Panel 1 : Effect of ELT on MCAS Scores after One Year of Implementation 
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Panel 2: Effect of ELT on MCAS Scores after Two Years of Implementation 
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Exhibit 5: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation Year 

Panel 3: Effect of ELT on MCAS Scores after Three Years of Implementation 


0 

N 

O) 

o 

£ 

u— 

LU 


LU 

U— 

O 

O 

0 


0.15 


0.1 


0.05 


lu -0.05 


- 0.1 


-0.15 


□ ELA 

□ Math 

□ Science 


Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 


Panel 4: Effect of ELT on MCAS Scores after Four Years of Implementation 
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EXHIBIT READS: After the first year of ELT implementation, students in ELT schools, on average, score 0.07 standard 
deviations below what they were predicted to score in the absence of ELT on the 3 rd grade ELA portion of the MCAS test. This 
difference was not statistically significant. 

Note: Full tables of these findings are presented appendix table F.3. Results are not shown for 8th grade science after one year 
because the estimated difference was 0.00. 

* statistically significant at p < .05 level 

** statistically significant at p < .01 level 

*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Sample: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. The sample sizes 
vary by cohort, implementation year, subject, and grade. The minimum number of schools is 29 and the maximum number of 
schools is 38. The minimum number of test records is 17,293 and the maximum number of test records is 39,242. 
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